It's possible to tell someone very assertively to leave you alone without having to resort to rudeness. I'm not sure what your situation is, but when I was younger and rode busses & trains really often, I used to have to deal with intrusive strangers on a fairly regular basis. I found that always having a book to read cut the unwanted attention dramatically, and when that failed (you would be surprised at the number of jokers who would plop down next to me on a practically empty bus and ask "Whatcha readin?") I used to just flat out tell people that I didn't want to talk and that did the trick 9 times out of 10. When it comes to that one person in ten who thinks they can force you into a conversation you don't want to have, look them steadily in the eyes and in a calm but firm tone of voice say something like, "Hey, I just told you I don't want to talk. That means I don't want to talk, not to you, not to anyone. I wanna sit here and read my book (or text, or play a game on my phone, or listen to my music, or do nothing at all, whatever.) Please leave me alone." Then turn your attention back to what you were doing before the person tried to engage you. If the person wants to call you some names or act like you're being a jerk, let them, especially if they do so as they're moving away from you.
It's almost always better to be assertive than to be rude, especially if you are a woman and the person forcing their conversation on you is some strange man. A lot of guys have this thing where they think that as long as they are being reasonably polite and not overtly sexual they should be able to strike up a conversation with any woman simply because they happen to be in the same place at the same time as she is, and furthermore, they seem to think that women have some sort of obligation to engage them in conversation if that's what they want. I assume that this is their perspective because of the number of times I have looked up from my book, told some complete stranger that I didn't want to talk, and been called some very nasty names and even threatened once or twice. If someone gets loud, belligerent or is frightening you with their words or behavoir, don't be afraid to reach out for help. Often, calmly but sternly telling the person "You are scaring me. All I want is to be left alone. Please respect my privacy." is enough to get them to see that they're behavior is inappropriate and make them back off. When that doesn't work, move away from that person and towards other people. Always stay around other people in such a situation. If you notice that you and the wierdo who has been harassing you are the only 2 people who get off the bus/train at your stop, turn around and hop back on the bus/train, ride it down a few more stops and take a different rounte home if you can. If you can't or if you don't notice that your harasser got off at your stop until it's too late to get back on the bus, act as though you're talking to someone (about calling the police, perhaps) on your phone, or go ahead and call 911 if the person is really creeping you out.
The truth is, no one has the right to force you to have a conversation with them anymore than they have a right to subject you to any other unwanted contact, and you are perfectly within your rights to refuse to be drawn into a conversation with anyone you do not wish to speak to. It really isn't a good idea for a woman to allow herself to be drawn into a conversation with a total stranger anyway, especially if that total stranger is a man. It's just really not a good idea at all to get too chatty or friendly with strangers, you don't know who they are or what they are capable of and it's better to be thought a cold, arrogant b***h than it is to be harassed, assaulted or even worse because you were afraid of sounding rude.