The surprise move, announced on Tuesday evening, is part of a crackdown on corruption and illegal cash holdings.
Banks will be closed on Wednesday and ATM machines will not be working.
India is overwhelmingly a cash economy. New 500 and 2,000 rupee denomination notes will be issued to replace those removed from circulation.
"Black money and corruption are the biggest obstacles in eradicating poverty," Mr Modi said.
People will be able to exchange their old notes for new ones at banks over the next 50 days but they will no longer be legal tender.
The announcement prompted people across the country to rush to ATMs that offer 100 rupee notes in an attempt not to be left without cash over the next few days.
Bribes are most often paid in cash so removing high value notes makes it somewhat more difficult to stuff money into an envelope. For a country like India where the government needs tax income to fund social and infrastructure projects, the war on cash is an understandable project. Even the ECB has committed to stop printing €500 notes by the end of 2018 because of the role these high value denominations have in allowing money to be moved around often for nefarious activities.
Banks are an obvious beneficiary of this move not least if they can promote the use of mobile banking apps, in the same way that China has, to reach people living beyond the branch network.
The Bombay Banks Index bounced impressively today from the region of its trend mean and a sustained move below 21,000 would be required to question medium-term scope for additional upside.
The Rupee was one of a small number of currencies that strengthened against the Dollar today and a sustained move above INR67 would be required to question Rupee demand dominance.