Yes, it's all binary. I believe there were a few experimental computers that used other system, but everything else is binary.

I should point out that outside of computer chips not everything is "binary" in the sense of a signal can be only in one of two states. For instance Ethernet cables can pass many different signal states. How many states depends on the speed of the connection, but it might be something like 64 or 256 different states. This is what makes it so fast. Instead of sending just a 1 or 0, you might send a "34", which actually means 01101, for example, so you are sending five binary values at once instead of just one. I'm hand waving a bit, but that's the idea. When them signal comes into your computer, it ultimately ends up being 0s and 1s again.

Short answer: something like a punch card

The first pieces of silicon weren't that (functionally) different than vacuum tubes. They were harder/better/faster/stronger, but still just a series of gates that could be turned on and off. In this way, there weren't really "programmed" in the sense that you would do something to them that would physically change the silicon. To use them, there are a bunch of pins that come out of the silicon, and you have to either put power on them or ground them to get it to do what you want. The easiest way to do this is to punch holes into pieces of paper and run those through a machine that detects holes or not holes and puts power on the pins accordingly. This is a punch card.

Eventually we had things like silicon memory (RAM) that could actually store information in the silicon (as long as the power was on), but these still stored programs (which is just the sequence of on/offs that the punch cards provided) externally - for example on magnetic tapes. Nowadays we have things like flash which do permanently store those values in silicon. To get the code into that silicon we use.... another computer that already has the code in it. To get it into that computer we use.... an earlier computer. If you follow this back all the way, you will end up with punch cards, basically.