I worked for Hollywood video when I was like 17, mixups like this were pretty normal. What would happen is that a video would be turned into the box and put back on the shelf without going through the computer, thus showing that the video was never returned. At some later time, another customer would try and rent that video, and it would show up in the computer as still rented out. Then a manager would have to come out and enter the video as FOS, for Found On Shelf, which would negate any late fees an such that had accrued on the last renter's account. The real problem would be if it was an unpopular video, then it could be some time before the mistake was caught during an inventory or from someone else trying to rent it.
Also, beyond the whole "their business model got outdated" thing, video stores wasted so much time and money on mundane stuff that had no bearing on their profitability that I wasn't at all surprised to see them start to go under. We used to have a district manager that would inspect the stores in his district to make sure that we weren't wearing white socks with black shoes, no shit. Same guy nearly wrote me up one time for wearing black leather cowboy boots instead of black leather loafers, like people were going to take their business to Blockbuster because I wasn't wearing loafers. Any business that's paying a manager's salary to a guy doing sock checks isn't going to last all that long, IMHO. In a publicly held corporation, I believe they may even call that sort of thing fiduciary misconduct. Netflix just put the final nail in the coffin, they were going to go anyway period.