>So if a government wants to borrow money it issues bonds at a fixed interest rate at a certain maturity date?

Correct. There are other ways a goverment can raise money, but issuing bonds is often seen as a good way to be able to spend more money without causing inflation. A bond is nothing more then "you pay us X now, we will pay Y back in Z years."

>How does it know what to set that rate at?

Goverments have credit ratings (similar to companies). Since a lot of investors use these ratings to decide to buy or not buy their bonds, goverments can use a similar analesys the investors use to decide there interest. Most goverments also have an institution to monitor there economy. These will probably have an inestigation ready with the effects of different interest rates.

>Bonds are sold on the secondary market at market value but that is just between private investors right?

Correct

>How does the secondary market price impact how the government services its debt given they will just offer par 100 at 4%?

The only thing that changes is who the goverment has to pay at the end of the agreed period. The secondary market doesn't change the bond, just the owner. Where it does matter is when they want to issue extra bonds. If on the secondary market your bonds are valued less then expacted new bonds will probably need a higher interest rate to be sold. While if there sold fir more they could lower the rate for New bonds.