The Federal Reserve extends credit in the form of short-term loans to member banks. Banks avoid taking loans from the Fed if they can, because it is viewed as a sign of instability.
The Federal Discount Rate applies to loans taken from what is known as the discount window at the Fed, and it tends to be a higher rate than what is charged between two banks. The Federal Reserve will extend credit only to banking institutions that are members of the Federal Reserve system.
Loans are made trough what is (somewhat sarcastically) known as the Discount Window (as if it were a teller window), and the reason this terminology is a little negative is because the general sentiment among bankers is that 99% of the time a bank who goes to the Fed for a loan is unstable, and was probably unable to get approved for a loan anywhere else.
While this may sometimes be true, it is also true that various factors, such as the fact that the Fed is willing to make such loans last-minute, when there may not be time to procure a loan from other institutions, sometimes plays a role. Fed credit can go towards Primary, Secondary, or Seasonal loans.
Seasonal loans are made to banks in areas that are affected by seasonal fluctuations in business, such as areas dominated by agricultural business or seasonal vacation destinations. Federal discount window loans are very short-term in nature, and are generally repaid within 90 days.
The interest rates on these loans is slightly higher than the Federal Funds Rate, and this is by design. The Fed would prefer that member banks do business with each other and that the rates are determined by the market forces of supply and demand. The Fed also requires approved collateral for such loans, such as marketable securities, which is not required in loans between banks.
The rate the Fed charges is the Federal Discount Rate, while the rates between banks is called the Federal Funds Rate.
What is the Federal Reserve Bank?
What is the Federal Reserve System?