How does TSA know which bags will be checked and which will be carry on?

TSA doesn't know. That is determined when you reach the ticket counter for your airline at the airport. 

For your carry-on baggage, you take it with you to the security checkpoint where it is screened. Bags selected for an inspection is pulled from the line and sent to an inspection station. Once the inspection is complete it is returned to you.

For your checked baggage at most airports, the airline agents will place a bag tag on baggage and place it on a conveyor system. The conveyor system will transport the baggage where it is automatically loaded into the x-ray system and unloaded back to the conveyor. Bags selected for inspection is re-routed to an inspection station. Once the inspection is complete it is returned to the conveyor system.

The bags that you present at the TSA point are the bags you're carrying on. Your other bags have left your control long before.

Because checked in bags are checked in before you go through TSA screening 

Yes. The carry-on is the one you are carrying with you when you go through the security checkpoint before you board the plane, which you do after you have given your checked luggage to the airline staff. The checked luggage goes to the baggage handlers after you check it at the airline counter. You don't see it again until you get off the plane. So it's easy to tell which is which. The carry on is what people go through security with.

Bags are checked at the airline counters before you go through the TSA checkpoints. The checked bags then go through the baggage handling system at the airport and carry ons are carried through the TSA security checkpoint by the passengers.

You don't take your checked bags through the TSA checkpoint.  You check them at the airline counter before you go through security, so you will only have your carry-on bags with you at that point.

Also, TSA doesn't determine what bags you can and can't bring on board.  They determine what items in those bags are or are not allowed.  

The airline staff decides if you have too many bags, bags that are too large, etc. 

What do you mean? You check your bag at the airline stand before you ever go through the TSA check. Sometimes they gatecheck- put the bigger carry ons in the hold, but that's not TSA. 

You hand over your checked bags before you get to TSA, not after.  

They ask you at check in. For some airlines, you have to announce choices when you book a flight.