From our good friend wikipedia
Most words ending in an unstressed -our in the United Kingdom, Ireland, New Zealand, Canada and Australia (e.g., colour, flavour, honour, neighbour, rumour, labour) end in -or in the United States (e.g., color, flavor, honor, neighbor, rumor, labor). Wherever the vowel is unreduced in pronunciation, this does not occur: contour, velour, paramour, troubadour, are spelled thus the same everywhere, with "contour" being an important technical term in mathematics and meteorology. Most words of this category derive from Latin non-agent nouns having nominative -or; the first such borrowings into English were from early Old French and the ending was -or or -ur. After the Norman Conquest, the termination became -our in Anglo-French in an attempt to represent the Old French pronunciation of words ending in -or, though color has been used occasionally in English since the fifteenth century. The -our ending was not only retained in English borrowings from Anglo-French, but also applied to earlier French borrowings. After the Renaissance, some such borrowings from Latin were taken up with their original -or termination; many words once ending in -our (for example, chancellour and governour) now end in -or everywhere. Many words of the -our/-or group do not have a Latin counterpart; for example, armo(u)r, behavio(u)r, harbo(u)r, neighbo(u)r; also arbo(u)r meaning "shelter", though senses "tree" and "tool" are always arbor, a false cognate of the other word. Some 16th and early 17th century British scholars indeed insisted that -or be used for words of Latin origin (e.g. color) and -our for French loans; but in many cases the etymology was not completely clear, and therefore some scholars advocated -or only and others -our only.
Webster's 1828 dictionary featured only -or and is generally given much of the credit for the adoption of this form in the United States. By contrast, Dr Johnson's 1755 dictionary used the -our spelling for all words still so spelled in Britain, as well as for emperour, errour, governour, horrour, tenour, terrour, and tremour, where the u has since been dropped. Johnson, unlike Webster, was not an advocate of spelling reform, but selected the version best-derived, as he saw it, from among the variations in his sources: he favoured French over Latin spellings because, as he put it, "the French generally supplied us." Those English speakers who began to move across the Atlantic would have taken these habits with them and H L Mencken makes the point that, "honor appears in the Declaration of Independence, but it seems to have got there rather by accident than by design. In Jefferson?s original draft it is spelled honour." Examples such as color, flavor, behavior, harbor, or neighbor scarcely appear in the Old Bailey's court records from the 17th and 18th century, whereas examples of their -our counterparts are numbered in thousands. One notable exception is honor: honor and honour were equally frequent down to the 17th century, Honor still is, in the U.K., the normal spelling as a person's name.
Derivatives and inflected forms. In derivatives and inflected forms of the -our/or words, in British usage the u is kept before English suffixes that are freely attachable to English words (neighbourhood, humourless, savoury) and suffixes of Greek or Latin origin that have been naturalized (favourite, honourable, behaviourism); before Latin suffixes that are not freely attachable to English words, the u may be dropped (honorific, honorist, vigorous, humorous, laborious, invigorate), may be either dropped or retained (colo(u)ration, colo(u)rise), or may be retained (colourist). In American usage, derivatives and inflected forms are built by simply adding the suffix in all environments (favorite, savory, etc.) since the u is absent to begin with.
Exceptions. American usage in most cases retains the u in the word glamour, which comes from Scots, not Latin or French. "Glamor" is occasionally used in imitation of the spelling reform of other -our words to -or. The adjective "glamorous" omits the first "u". Saviour is a somewhat common variant of savior in the United States. The British spelling is very common for "honour" (and "favour") in the stilted language of wedding invitations in the United States. The name of the Space Shuttle Endeavour has a u in it since this spacecraft was named for Captain James Cook's ship, the HMS Endeavour.
The name of the herb savory is thus spelled everywhere, although the probably related adjective savo(u)ry, like savour, has a u in the U.K.. Honor (the name) and arbor (the tool) have -or in Britain, as mentioned above. As a general noun, rigour (/ˈrɪɡər/) has a u in the U.K.; the medical term rigor (often pronounced /ˈraɪɡɔr/) does not. Words with the ending -irior, -erior or similar are spelled thus everywhere and have never had a "u", for example inferior or exterior.
Commonwealth usage. Commonwealth countries normally follow British usage. In Canada -or endings are not uncommon, particularly in the Prairie Provinces, though they are rarer in Eastern Canada. In Australia, -or terminations enjoyed some use in the 19th century, and now are sporadically found in some regions, usually in local and regional newspapers, though -our is almost universal. New Zealand English, while sharing some words and syntax with Australian English, follows British usage.