To give an analogy for those of you who use Twitter, it is very similar to the following concept of Twitter. You are expressing an interest in their updates. Likewise, with a blog.
You can follow a blog either publicly or privately. As their names suggest, following a blog publicly means that other people are able to view your interest as it will appear on the blog home page. Following privately means that you can still keep up to date with the blog, but your interest in it is private, and so you do not appear on the blog home page.
To follow a blog is simple. All you need to do is click the 'Follow' or 'Join this site' button (the exact wording may be different between different blogs).
You may also keep up to date with a blog by subscribing to it. On my blog to the right, you are able to subscribe to the posts and/or the comments, the choice is yours. At the bottom of the page, you can also subscribe to the blog using an Atom feed. This is simply an RSS (Really Simple Syndication) feed.
Most bloggers, including myself, will also subscribe to other blogs. This is called the blog roll. My blog roll is located on the right hand side, under the heading 'Blogs that I follow' (as not everyone will know what a blog roll is). These are the blogs that I follow, and have an interest in. They cover a wide range of topics and subjects, from software development, to atheism, to animal welfare to name a few.
So if you come across a blog that interests you, be sure to follow it. It lets the blog author know that what they are writing about is considered interesting by other people, and that you value their efforts. I would also encourage people to leave comments on blogs.
So there are several ways to keep up to date with your favourite blogs. Go ahead and give them a try.