Staff decide on board
Conductors can ask you to move your bag to the oversized baggage area and may charge a penalty if they judge it too large. There is no guaranteed outcome, it depends on the situation.
Fix it before boarding
If you realise at the station, change your reservation at a ticket machine or JR desk to a seat with an oversized baggage area. It is usually quick if seats are available.
Travelling with a Japan Rail Pass? Seats with an oversized baggage area are included at no extra cost. Get your Japan Rail Pass