With a vegetable peeler, remove 8 strips of orange peel, each 2 inches long. Try not to remove any of the white pith. If there is white pith, scrape it off with a paring knife.
Bring a large pot of salted water to a boil over high heat. Add fresh fennel and cook for 3 minutes. Remove from heat and drain.
Warm olive oil in a large saucepan over medium heat. Add orange peel, fresh fennel, fennel seed, garlic and chile flakes, and cook until all begin to sizzle, about 1 minute. Add olives and warm gently for 5 minutes. Remove from heat and let sit 6 hours. Discard orange peel.
Ten minutes before serving, rewarm olive mixture. To serve, place olives and fennel on a small platter. Drizzle with a few tablespoons of the flavored oil and serve immediately.
Adapted and reprinted with permission from "Wine Country Cooking," by Joanne Weir, copyright 1999, 2008, published by Ten Speed Press, an imprint of Penguin Random House LLC. Photo by Richard Jung.