The index of sentiment derived from a monthly Reuters survey of manufacturers rose by 3 points to plus 16 in August, which matched the level it was at in November 2010. A positive readings shows optimists outnumbered pessimists.
The index is expected to rise again to plus 18 in November in the Reuters survey, which is strongly correlated with a closely watched quarterly Bank of Japan tankan survey.
Makers of cars and electronics, however, were less upbeat due to worries about slowing growth in emerging markets, highlighting the risk overseas economies pose to Prime Minister Shinzo Abe's reflationary policies.
"Sales are doing much better than we initially anticipated," one chemicals maker said in the August 2-9 poll of 400 big and medium-sized firms, of which 276 responded.
The service sector gauge also rose 5 points in August from the previous month to plus 23. This marked the highest level since April 2007 due to a housing boom and increased corporate spending on IT services.
The index is expected to improve further to plus 29 in November.
The BOJ's tankan issued on July 1 showed manufacturers' sentiment turned positive in April-June for the first time in nearly two years and was seen rising further on the feel-good mood generated by Abe's policies, intended to end 15 years of deflation, spur hiring and improve wages.
The BOJ will release its next tankan survey on October 1.
The central bank kept monetary policy steady earlier this month and left its economic assessment unchanged as board members wanted more time to measure the strength of capital expenditure.
Comments from the Reuters Tankan survey in August suggest that a turnaround in business investment is still distant.
"Europe's economy is in the doldrums and domestically companies feel they still have excess capacity," said one machinery maker.
"Going through with new capital expenditure is much more difficult than we thought."
Japan's economic growth slowed more than expected in the second quarter, driven by an unexpected fall in capital expenditure in a worrying sign of companies' reluctance to invest.
Strong consumer spending, public works spending and a rebound in exports suggest that Japan's economic recovery can easily remain on track.
However, investor outflows from emerging markets due to worries about deteriorating growth could become a more of a concern for Japanese companies and policymakers.
(Editing by Eric Meijer)
Japan Manufacturers' Optimism Hits Three-year High