Plaintiff Spencer Harris, an inmate at Menard Correctional Center in Illinois, appeals from the magistrate judge's entry of summary judgment in favor of defendants2 in this civil rights action filed pursuant to 42 U.S.C. § 1983.

This case returns to this court after our previous remand on the claim that the defendant prison officials had retaliated against plaintiff, both in job terminations and cell transfers, for his law-related activities. Harris v. Fleming, 839 F.2d 1232, 1238 (7th Cir.1988). After remand, the magistrate judge reconsidered his previous denial and appointed counsel for plaintiff. Harris, 839 F.2d at 1238 (suggesting reconsideration of appointment of counsel). Subsequently, defendants again moved for summary judgment, to which plaintiff failed to respond. The magistrate judge granted summary judgment in favor of defendants. On appeal, Harris maintains that summary judgment was improper because the record shows genuine issues of material fact remain on the question of whether defendants had retaliated against him for exercising his constitutional right of access to the courts. He also argues that the magistrate judge should have granted his request for a substitution of appointed counsel.

We review a grant of summary judgment de novo. Cooper v. Lane, 969 F.2d 368, 370 (7th Cir.1992). Summary judgment is proper where there is no genuine issue as to any material fact and the moving party is entitled to judgment as a matter of law. Fed.R.Civ.P. 56(e). The court must review the record, and reasonable inferences therefrom, in the light most favorable to the party opposing the motion. Gomez v. Chody, 867 F.2d 395, 398 (7th Cir.1989). The nonmovant, however, must affirmatively demonstrate by specific and supported facts that there is a genuine issue of material fact. Celotex Corp. v. Catrett, 477 U.S. 317, 322-26 (1986). Nevertheless, " '[w]here the moving party fails to meet its strict burden of proof, summary judgment cannot be entered even if the opposing party fails to respond to the motion.' " Cooper v. Lane, 969 F.2d at 371 (emphasis added), quoting Big O Tire Dealers, Inc. v. Big O Warehouse, 741 F.2d 160, 163 (7th Cir.1984). It is always prudent, however, to respond to a summary judgment motion. Cooper, 969 F.2d at 371.

An act taken in retaliation for the exercise of a constitutionally protected right may give rise to a claim under § 1983. Rakovich v. Wade, 850 F.2d 1180 (7th Cir.), cert. denied, 488 U.S. 968 (1988); Matzker v. Herr, 748 F.2d 1142 (7th Cir.1984). The right of access to the courts is one of the fundamental rights protected by the Constitution. Bounds v. Smith, 430 U.S. 817, 821 (1977); Corgain v. Miller, 708 F.2d 1241, 1247 (7th Cir.1983). Prison officials may not retaliate against an inmate because of his exercise of the right of access to the courts. This principle is applicable even where the action taken in retaliation would be otherwise permissible. Cain v. Lane, 857 F.2d 1139, 1145 (7th Cir.1988); Howland v. Kilquist, 833 F.2d 639, 644 (7th Cir.1987). Retaliation can be inferred from a chronology of events. Cain v. Lane, 857 F.2d at 1145; Murphy v. Lane, 833 F.2d 106, 109 (7th Cir.1987); Benson v. Cady, 761 F.2d 335, 342 (7th Cir.1985). While merely alleging the ultimate fact of retaliation is insufficient, a court may draw inferences from a chronology of events where direct evidence of retaliatory intent is lacking. Benson v. Cady. To prove retaliation, plaintiff must show that the protected conduct was a substantial or motivating factor in the decision and that the same decision would not have been reached absent the protected conduct. Mount Healthy City School District v. Doyle, 429 U.S. 274, 287 (1977).

At the time of the first summary judgment in 1986, the record showed:

"In June 1985, according to Harris, an Assistant Attorney General visited him at Menard seeking to settle his pending lawsuit against Menard. Harris declined to settle, and alleges that the defendants knew of this refusal. Harris says that he was fired from this third job as a foodhandler the next day, and that two days later he was transferred against his will into a cell with a known homosexual. Harris claims that he was given no reasons for those actions, and the defendants have not denied that they acted in retaliation. The timing of his firing and transfer, he argues, is enough to make retaliation a jury question.

Prior to these incidents, in December 1984, Harris argues, he held two prison jobs, library clerk and salesman for Lifer's Inc. That month Harris achieved a remand from this court of his prior suit against Menard personnel. [ Harris v. Greer, 750 F.2d 617 (7th Cir.1984) (reversing Magistrate Judge Cohn's decision).] Within about a week of the remand he lost both jobs. Harris was told that there was inadequate security for him in his work for Lifer's Inc., and that he was fired from the library job because he filed a monthly library form a day late (even though the prison was on lockdown at the time). Harris disputes these reasons as 'not ringing true,' pointing out that when security again reached adequate levels, he was not restored to his Lifer's Inc. job. The library job, Harris says, was important to him in the pursuit of his litigation.

Harris has sworn that he was transferred from cell to cell about every six weeks, often enough to prompt the prison psychiatrist to inquire as to the reason. We do not find the answer the psychiatrist may have received. Other prisoners were not so regularly rotated. Ultimately Harris was transferred into the general population against his wishes and, he claims, housed in a cell block with several known enemies of his." Harris, 839 F.2d at 1236-37.

In support of their motion for summary judgment after remand, defendants filed three affidavits. Frailey submitted an affidavit stating that Harris was removed from his job as a food handler "after it was brought to my attention that he was in C-grade," because inmates in C-grade are not permitted assignment."3 Brieschke filed an affidavit stating that Harris was discharged from the library job on or about November 1984, due to "poor performance of his duties (he failed to submit monthly reports and to maintain appropriate records)." While Brieschke uses the plural ("reports" and "records"), Fleming indicates there was only a single report missing. Fleming filed an affidavit stating that on or about November 1984, plaintiff was removed from his library clerk assignment "for failure to submit a monthly report." Fleming also stated that the "detail as a Lifer's Inc. salesman is not an institutional assignment." The Lifer's Board "holds the authority for removing inmates from their assignments with Lifer's Inc." None of the affidavits indicate the information was within the personal knowledge of the affiant, or the source of their information. See Fed.R.Civ.P. 56(e); Pfeil v. Rogers, 757 F.2d 850 (7th Cir.1985), cert. denied, 475 U.S. 1107 (1986). See also Cummings v. Roberts, 628 F.2d 1065 (8th Cir.1980). Moreover, no prison records are attached to the affidavits as support for the conclusory statements made in the affidavits.

In his 1991 order granting defendants' motion for summary judgment after remand, the magistrate judge found that defendants' affidavits indicated that the firings were justifiable or not their responsibility. The magistrate judge concluded:

"Plaintiff has alleged numerous acts of retaliation in this cause of action, but has failed to indicate a chronology of events necessary to infer retaliation. That plaintiff was discharged from his prison jobs as the result of his actions is quite evident. Plaintiff's conclusion that the firings were in retaliation are not sufficient to continue this cause of action."

There were no further details given as an explanation of the magistrate judge's decision. Thus, this case is in no better posture to support the grant of summary judgment now then when it was before us last in 1988. See Black v. Lane, 824 F.2d 561, 562 (7th Cir.1987) (reversing summary judgment for defendants in inmate's racial discrimination suit). See also Smith v. Maschner, 899 F.2d 940, 948 (10th Cir.1990) (summary judgment for prison officials reversed where discipline taken immediately after inmate's return from court, prison officials had never before taken similar action against inmate, and punishment was sizeable given his prior good record).

Nor does the second motion for summary judgment, or the magistrate's judgment, deal with the numerous cell transfers which we previously stated also supported a claim of retaliation. Despite our attention to this factual issue in our earlier opinion, the new affidavits filed by defendants in support of their 1991 motion for summary judgment say nothing about cell transfers. They only discuss the job terminations. Also, on remand the magistrate judge made no factual findings in regard to the cell transfers. He did not consider them within the context of the chronological pattern. Instead, he simply noted that prison officials have broad discretion in making housing decisions, and there is no liberty interest in remaining within a particular prison. We have already found that the issue here is not whether Harris has a "due process right to job or cell assignment." Harris, 839 F.2d at 1238.

We conclude that the chronology of events relating to both the job terminations and cell transfers, as shown from the documents in the record on appeal, supports an inference of retaliation which is sufficient to withstand a motion for summary judgment at this point in the proceedings. See Smith v. Maschner, 899 F.2d at 949 ("Where defendants' motives are seriously at issue, trial by affidavit is particularly inappropriate"; credibility determinations and the weighing of evidence is for the jury).

Defendants rely heavily on the fact that, on remand, when they filed a second motion for summary judgment with three accompanying affidavits, neither plaintiff nor his appointed counsel filed anything in response. Plaintiff, appearing pro se in this appeal4, counters that he had repeated difficulties trying to communicate with appointed counsel, and that counsel never informed plaintiff that defendants' pending motion for summary judgment had been filed until after the district court heard the motion, and that in any event plaintiff was "unaware and not informed that he had to answer the defendants' affidavits." As we advised in another case, it is always prudent to respond to summary judgment, but lack of a response does not mean defendants are entitled to summary judgment as a matter of law. Cooper v. Lane, 969 F.2d at 371.

The record belies the magistrate judge's cursory finding that "performance was reviewed and found to be satisfactory."5 See Lewis v. Lane, 816 F.2d 1165 (7th Cir.1987) (reversing Magistrate Cohn's grant of summary judgment for § 1983 defendants and against inmate plaintiffs on basis that substitute counsel should be appointed and given the opportunity to file counter-affidavits in opposition to defendants' summary judgment motion). Appointed counsel failed to file a written response and nothing in the record suggests that counsel took any action to further Harris's claim (such as conducting discovery). Moreover, the magistrate judge was aware that appointed counsel had obtained numerous extensions of time to file an amended complaint which was not done until the court issued an order to show cause why the case should not be dismissed for failure to prosecute. This pattern should have signalled the magistrate judge that the failure to file a response to the summary judgment motion was due to counsel's "continued lack of diligence rather than a strategic decision." Lewis, 816 F.2d at 1170. When a response to the summary judgment motion was not filed, perhaps the better practice would have been to require that plaintiff get the notice required by Lewis v. Faulkner, 689 F.2d 100 (7th Cir.1982).

In view of the need to remand this case for the second time, and the tortuous road this case has taken, we direct that upon remand the district court appoint substitute counsel for Harris. We also direct, pursuant to Circuit Rule 36, that the case shall be assigned to a different judge. We do not intend to indicate in any way whether the case can survive a summary judgment proceeding which complies with the holdings contained herein, or whether the case should be tried on the merits.

For these reasons, we hold that it was error to enter summary judgment in favor of defendants. The judgment of the magistrate judge is REVERSED. The cause is REMANDED for reassignment to a different judge, appointment of substitute counsel, and further proceedings consistent with the holdings contained herein.