As an intermediate swimmer, you came to swimming as an adult. You learned to swim freestyle so you could race triathlons. You are beyond the beginner stage in that your 1500 meters to 2.4-mile freestyle times in races are consistent and comfortable—just not very fast.
Forget About Perfect Form
When most adults take on the task of learning how to swim, the main focus is on "perfect form." Coaches and instructors do swim drills as the majority of the workout, assuming swimming fitness and endurance will be a by-product of drills. This may be true, but only up to a point.
If you've trained in this drill-focused manner, I'm sure you found that you had great gains in swim fitness and speed early on. Then you reached a plateau in speed. Each time you try to swim faster your form seems to deteriorate, stroke count per 25 meters increases and you're gasping for oxygen when you reach the wall.
Yes, as you pick up speed your form will not be the same as when you are comfortably drilling your way through 1000 meters. Expect your stroke to be different when you swim fast.
When you try to swim faster, stop worrying about, and trying to have, perfect form.
Just Swim Fast
Good swimming technique is important, but if you watch any world championship meet on television or watch form videos on the internet, you will find that the fast swimmers do not have identical techniques and form. You need to develop the form that works best for you and your particular anatomy. That takes some time.
More than likely the top swimmers you know spent hours in the pool as kids. I'm willing to bet a good number of them spent a lot of time in the water just messing around. They played various games with no regard for anything other than swimming fast in a game of tag, seeing who could tread water the longest, diving after objects on the bottom of the pool, seeing who could swim the furthest underwater and various other water entertainment activities.
Sadly, as adults, few of us have the time to play in the pool for hours and hours. Time pinch is an issue and the fact is many of our friends are boring and don't want to play pool games. But you can apply the same philosophy to your workouts.
When you start working on fast swimming in the pool, just focus on going as fast as you can. Make it a game to see how fast you can swim. Start with short distances of fast swimming at the start of your workout once or twice per week.